1. Field of the Invention
The present invention relates to an image processing apparatus for realizing three-dimensional computer graphics and a method of the same, more particularly relates to a configuration of a computer graphics apparatus and method for generating more realistic images.
2. Description of the Related Art
Along with the increase in operation speeds and improvement of graphic drawing functions in recent computer systems, computer graphics (CG) for preparing and processing graphics and images by using computer resources has been actively researched, developed, and put into practical use.
For example, three-dimensional graphics generates more realistic, three-dimensional-like two-dimensional high definition images by expressing an optical phenomenon when a three-dimensional object is illuminated by a predetermined light source by a mathematical model, applying shading or gradation to the object surface based on this model, and further adhering patterns.
Such computer graphics is now being increasingly actively utilized in CAD/GAM in the field of development in science, engineering, and manufacture and other various fields of applications.
Three-dimensional graphics is generally comprised of a “geometry sub system” positioned at the front end and a “raster sub system” positioned at the back end.
The “geometry sub system” is the step for performing geometricerations on the location, posture, etc. of a three-dimensional object to be displayed on a display screen.
In a geometry sub system, generally an object is handled as a mass of a large number of polygons. “Coordinate conversion”, “clipping”, “light source computation”, and other geometricerations are carried out in units of polygons.
On the other hand, the “raster sub system” is the step of painting pixels composing the object.
Rasterization is realized by interpolating image parameters of all pixels included inside a polygon based on for example the image parameters found for the vertexes of the polygon.
The “image parameters” referred to here include for example color (generated color) data expressed by the so-called RGB format and z-values for expressing a distance in a depth direction.
Further, in recent high definition three-dimensional graphics processing, also an f (fog) for giving a perspective feeling and a texture t for imparting realness by expressing the feeling of a material and the pattern of the object surface etc. are included as image parameters.
Here, processing for generating pixels inside a polygon from vertex information of a polygon is frequently executed using a linear interpolation technique referred to as a “digital differential analyzer” (DDA).
In a DDA process, an inclination of the data to a side direction of the polygon is found from the vertex information, the data on the side is calculated by using this inclination, and then the inclination in the raster scan direction (X-direction) is calculated, and the amount of the change of the parameter found from this inclination is added to the parameter value of the start point of the scanning to thereby generate the internal pixel.
In such three-dimensional graphics, in recent years, in order to generate more realistic computer graphic images, the illumination model has been improved. On the other hand, research and development are being conducted into the hardware architecture for pursuing real time graphic generation performance.
In general, when trying to improve the realness, the amount of operations increases and the real time property is degraded, while when trying to improve the real time property, simplification of the algorithms is required and the realness is sacrificed. Research and development are being conducted to realize these conflicting requirements simultaneously as much as possible.
Among the newest hardware technologies for pursuing real time graphic drawing performance, there is the technology referred to as a “programmable pixel shader”.
A programmable pixel shader imparts the function of programmability to the step of shading performed for every pixel composing a polygon.
By this programmability, a general reflection model referred to as a “bi-directional reflection distribution function (BRDF)” has started to be mounted, and the possibility of the real time, realistic image generation has appeared. Further, it has also been considered to change the nature of the BRDF so as to impart change to the color of a polygon surface by texture mapping. Such a BRDF having a different nature according to the location on the surface of the polygon will be referred to as a “shift-variant BRDF”.
Below, a detailed explanation will be given of a programmable pixel shader, BRDF, and shift-variant BRDF.
Programmable Pixel Shader
A programmable pixel shader imparts the function of programmability to the step of calculation of luminance performed for every pixel.
The technique of the programmable pixel shader is employed in graphic LSIs of GForce3 and GForce4 of NVIDIA Corp. and RADEON 8500 etc. of ATI Corporation.
Using this, a user can devise and write a pixel shader program to realize a shading model having a realness not achieveable by conventional fixed hardware supporting only gouraud shading in a real time execution environment.
For example, when using not only the techniques of gouraud shading, texture mapping, etc. mounted also in conventional fixed hardware, but also the programmable pixel shader, Phong shading, bump mapping, toon shading, environment mapping, etc. can be mounted. Further, an anisotropic reflection model and a part of kind BRDF can be utilized.
When using a programmable pixel shader, a texture memory can be used as a place for storing shading parameters mapped on a polygon.
For example, when there is a shader program utilizing a normal direction, the data is stored in the normal direction in each texel of the texture memory. At the run time, shading can be calculated by using the normal obtained from the texel value. Bump mapping and partial BRDF have been realized by this method.
The mechanism of the programmable pixel shader of the CForce3 and GForce4 of NVIDIA Corp. is shown in FIG. 1.
Basically, this programmable pixel shader is configured by a texture lookup unit 1 and an ALU 2.
Four texture addresses TEX0 to TEX3 are supplied from an upstream programmable vertex shader to the texture lookup unit 1. The texture lookup unit 1 can look up the texels of four elements of RGBA corresponding to four texture addresses at the maximum. The lookup can be executed for four elements in parallel. The results of the lookup are output to the pixel ALU 2.
Further, the programmable vertex shader supplies four elements of two RGBA to the pixel ALU 2. The pixel ALU 2 has special operation units such as register combiners 2-1, 2-2, and 2-3 as shown in FIG. 2.
The register combiners 2-1 and 2-2 execute inner multiplication and multiplication, while the register combiner 2-3 executes addition. In the pixel ALU 2, the data is input to the register combiner 2-1 through input maps 2-4 and 2-5, and the data is input to the register combiner 2-2 through input maps 2-6 and 2-7. By this, the multiplication and the inner multiplication can be executed in parallel. Note that the inner multiplication is only three-dimension×three-dimension. The inner multiplication operation can be utilized for the inner multiplication of for example the Phong reflection model and the diffuse reflection model.
The pixel shader program is a program of a few steps which can be defined by the user and executed from the header for every pixel. This program controls the texture lookup unit 1 and the ALU 2.
BRDF
BRDF is a function expressing the ratio of the reflection of the light incident upon one point on the surface at the object surface. This is expressed as a four-dimensional function comprised of directions of the light source and the viewpoint as in the following equation:Io=BRDF(θi, φi, θo, φo)cos θiIi  (1)
The BRDF can be treated as a higher concept of the conventional well known shading models. For example, the following three lighting effects can be thought of as sub sets of BRDF using directions of the viewpoint and the light source as the parameters.
The ambient lighting model is one of the lighting effects determined independently from the directions of the viewpoint and the light source.
The diffuse reflection model is one of the lighting effects determined according to the direction of the light source.
The specular reflection model is one of the lighting effects determined according to the directions of the light source and the viewpoint.
Other than the above description, a shading model having anisotropy and a retro reflection model for reflecting the light to the direction of the light source etc. can be expressed.
In order to most simply express also a general purpose BRDF, four-dimensional data may be used. However, when trying to utilize it industrially, the cost of realization would be high and therefore this would not be realistic. The formats for expression of the BRDF are roughly classified into those of a physical equation, experiential equation, and approximation equation as shown below.
Physical Equation
This is an equation derived from physical principles. The operation is complex in many cases. For example, this includes a Torrance-Sparrow model, a Oren-Nayar model, etc.
Experiential Equation
This is an equation using compact expressions for reproducing conspicuous characteristics which can be perceived by human beings. Visually definite results are obtained. The processing cost is also low. For example, this includes a Phong shading model, polynomial texture maps (Tom Malzbender et al., “Polynomial Texture Maps”, ComputerGraphics (SIGGAPH2001 Proceedings), pp. 519–528), a Lafortune-BRDF (E. P. F. Lafortune et al., “Non-Linear Approximation of Reflectance Functions”, ComputerGraphics (SIGGAPH1997 Proceedings), August 1997), etc.
Approximation Value, Approximation Equation
This is an equation using various types of base functions and using tabularized data. It is a method considered to compress the data of a four-dimensional table provided in a general purpose BRDF. For example, it includes a Kauzt-BRDF (J. Kautz et al., “Interactive Rendering with Arbitrary BRDFs using Separable Approximations”, Proceedings of the 10th Eurographics Workshop on Rendering, pages 281–292, June 1999) model, Wavelet, BRDF using spherical-harmonics, etc.
The Lafortune-BRDF model is BRDF defined by using a generalized cosine lobe.I=ρs[Cxuxvx+Cyuyvy+Czuzvz]″  (2)
An anisotropic specular effect, retro reflection, off specular effect, etc. can be reproduced.
A polynomial texture map is a shading model specializing in its self shadow effect.I=a0lu2+a1lv2+a2lulv+a3lu+a4lv+a5  (3)
The parameters of the model can be obtained from a plurality of images captured by using a dedicated measurement device.
A Kauzt-BRDF model is a BRDF model approximating a four-dimensional table by multiplication of two two-dimensional tables by using normalized decomposition. There is mounting using the programmable pixel shader of NVIDIA Corp. The reproducibility of the feeling of a metal and plastic is good. However, two tables are used, so the required amount of data is large.
Shift-Variant BRDF
A BRDF changing according to the location on the surface will be referred to as a “shift-variant BRDF” or “BTF” (hereinafter, standardized as “shift-variant BRDF”). A shift-variant BRDF is a six-dimensional function. It is also possible to consider as if the two-dimensional parameters expressing the location on the surface of the polygon were used in the BRDF.
Since there are different BRDFs at different locations on the surface, a material having a nonuniform texture can be realistically expressed. For example, this is suitable for expressing clothing, fabric, or skin.
With a programmable pixel shader, by holding the parameters of the BRDF model in a circuit similar to texture mapping and looking up the parameters, the BRDF can be changed in units of pixels. Namely, a shift-variant BRDF can be realized.
For example, the Daubert-BRDF (Katja Daubert et al., “Efficient Cloth Modeling and Rendering”, in Rendering Techniques, '01 Proceedings of the 12th Eurographics Workshop on Rendering 2001, Springer Verlag, Berlin) experiments with making a shift-variant BRDF of the Lafortune-BRDF model.
However, the Daubert-BRDF only stores the parameters of Lafortune-BRDF equations in the texels for achieving shift-variance. The coordinate system for evaluating Lafortune-BRDF equations is not made shift-variant, so bump map-like expressions are not possible. An effect dependent upon the direction of the viewpoint is realized by using a cube map. In mounting using a cube map, this effect cannot be made shift-variant. Further, the self shadow effect is not considered at all.
A polynomial texture map calculates luminance by using a two-dimensional equation comprised of six terms as in the above equation 3.
This technique aims at replacement of conventional texture mapping. A table storing six coefficients of the terms in the texels may be used. The self shadow effect is a characteristic feature of this.
The programmable pixel shader, BRDF, and shift-variant BRDF explained above suffer from the following disadvantages however.
The programmable pixel shader imparts programmability to the pixel shader—which was conventionally a fixed hardware circuit—and imparts the flexibility of a general purpose operation unit.
However, in order not to detract from the operation efficiency—the sales point of graphics hardware, priority has been given to efficiency in generalization.
There are restrictions, however, such as the fact that branch commands cannot be used, the lack of orthogonality in the command set, and the limitation of the data type to 8 bits.
The programmable pixel shader mounted in the GForce series of NVIDIA Corp. has a circuit configuration assuming a shading model enabling calculation by looking up texels comprised of four elements from a maximum of four texture maps and several inner multiplication operations or multiplication operations for every pixel.
Various shading models can be mounted, but there are also shading models with poor mounting efficiency or unable to be mounted. Particularly, the processing efficiency of a shading model including many matrix operations is poor. Further, only 16 elements can be obtained at the maximum from four texture maps. This is insufficient for a sufficient supply of matrixes.
That is, it is necessary to predict a shading model which may be utilized in the future as correctly as possible and consider the hardware architecture of the programmable pixel shader having a good balance between efficiency and generality.
There are diverse forms of expression of a BRDF. Each has its characteristic features. The application may select the form of expression of the BRDF in accordance with the objective.
However, for graphics hardware designed giving priority to efficiency and considering the minimum generalization, desirably the corresponding BRDF model is the minimum and thereby obtains the maximum effect. A convenient BRDF model satisfying these contradictory requirements is required.
Further, in order to realize a good shift-variant BRDF, a high expression power, compact BRDF model becomes necessary.
The form of expression of a BRDF having a small number of parameters is suited for a shift-variant BRDF. When the number of parameters allocated to one texel is small, the size of the table becomes smaller by that amount. Conversely, when the number of the parameters is large, the table becomes larger, so a BRDF of a type obtained by combining Kauzt-BRDF or other tables is not suited for obtaining a shift-variant BRDF.